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ABSTRACT 

A brief exatrtination of research on the relationships 
among the train ing*of teachers, their classroom behavior, and the 
learning of children suggests a general framework for studying both 
the intermediate and ultimate effects of teacher training. The idea 
behind this model is that such effects are more plausibly 
demonstrated if the argument is broken down into several intermediate 
steps between training activity and child growth. Each stepUs 
claimed to' affect the next, and the demonstration addresses the truth 
of each of these claims. This booklet is intended to help the teacher 
examine the effects of training *ef forts in ways that lead to 
convincing demonstration of their effects. Outlines are presented on 
the following topics: (1) questions and answers about demonstrating 
effects; (2) evidence that demonstrates effects; (3) planning the 
demonstration of effects; (4) judging the evidence; (5) a model for 
demonstrating training effects; (6) measurement issues in 
demonstrating effects; and (7) sources and resources on demonstrating 
effects. (JD) 
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WHAT THIS IS ABOUT 



y 



All of us look for new ideas and better ways of doing -things. And 
often, we are forced to try a new jdea ourselves to find out if it works 
because there is no evidence about its effects* There are* many reasons 

*why we don't demand such evidence for the new ideas we try, and many 
excuses for not collecting such .evidence for our own ideas — we don't 

• have time, it can'J be dorfe, no pne would listen anyway, the effects of 
what we do cannot be joneasured* Traditional approaches to educational 
.evaluation,, with their high standards and high costs, and with their 
emphasis on scientific rigour have given us good reason to be sceptical 
about* demonstrating effects. 

We neeci to recognize that some evidence which demonstrates 
effects is better than none at all* We need to recognize that each of us 
can collect some information no matter what our constraints on resources 
and expertise. *If our efforts fall far short of what the evaluation experts 
would recommend, they still may yield results of substantial worth to us* 

, % These materials are intended to help you examine the effects of 
your training efforts in ways that lead directly to a convincing 
demonstra.tion of their effects, Please' give us # your feedback, using the 
self-mailing feedback card included in the ^package," to help us reach this 
objective. , - J • ' ' ^ 



We hope*you find these materials interesting and useful 
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QUESTIONS AND ANSWERS ABOUT DEMONSTRATING EFFECTS 



What is a claims-based demonstration of effect? ' A demonstration of effects presents 
information about an educational practice which a'rgues that it has Iliad certain impacts 
on those it serves. Spfeoific claims are made that the practice has reliable effects, and 
evidence is presented to Convince a sceptical audience that these claims are indeed 
true. Claims of effectiveness are initially selected from the beliefs pf, program staff 
and others about the effects of the % practice. Existing information is then reviewed to 
see if there is sufficient evictencq to support e&ch claim. WKfere existing evidence is 
not convincing, then new evidence may%e gathered or the claim may b$ dropped from, 
the demonstration. The final demonstration consists' of a bhef document that succinctly 
describes the practice, states the claims, and summarizes the evidence supporting each 
claim. ^ y ' 

What is a claim^ased demonstration of effects good for? An educational practice 
should be .examined to see if it works before it is widely disseminated. By spelling out 
clearly what a practice claims to do and by giving convincing evidence * that the 
claimed effects actually occur, a developer shows the' market that its' innovative 
practices are worthy of adoption. In addition to supporting dissemination, a 
demonstration of effects is useful in showing funding agents and other sources of 
support that a development effort has been successful. Also, 'an attempt to demonstrate 
effects often reveals weaknesses in an evaluation design by highlighting gaps between 
the claims a staff would like to make and the evidence t 'that is actually available. 
Finally, as a succinct statement of impact, a claims-based demonstration may be used 
to ^uolicize, the accomplishments of a project rn any number of Wcfys. * 

Does the depionstration of effects require rigorous scientific evaluation? The purpose 
of a demonstration of effects is to convince an afudiemte that a' practice is effective, 
and many types of evidence can be helpful in Achieving this purpose: High quality 
evidence^^eollected through carefully planned evaluations may be more convincing, but 
it is* not required to demonstrate effects. ' It is more important, to make some 
demonstration using whatever evidence is available than to insist on high standards of 
scientific validity. Once claims are identified and available evidence assembled, 
evaluation processes can be^irftproved f to make the demonstration ^more convincing. 

Must a demonstration prove that the practice caused the observed effects? The desired 
effects of most training are somewhat removed from the training -itself, and herjce 
many other factors>can affect the relationship between a training practice and its 
intended effects. Thus it is impossible to conclusively prove" that a training practice Is 
the sole cause of an observed effect; at best we cap rul.e otjt some of the most 
plausible alternative explanations for the effects' we observe. But .what is'*rr\ost 
important is to 'determine if the intended effects indeed occurred. / 
4 * / . 

r • * 

How much time does this all take? An initial demonstration based on evidence you 
already have can 'be made in about two person-days using the process described above. 
' The result is a summary like the examples included below. If the demonstration needs 
to be improved by collecting new*evidence, more time is required to develop* or locate 
measurement tools, foV collection of evidence, and for its analysis. Other things being 
equal, the more time spent, the more the demonstration is improved. But note that 
often evidence to improve a demonstration may take only a modest amofint of time to 
collect. ^ 
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EVIDENCE TH^T DEMONSTRATES EFFECTS 



M^nynypes of evidence^ can demonstrate effects. Some of the more common types are 
lfsted below: t . 

o test trainees before and after the trailing to see if they learned 'basic 
information or acquired important Cognitive skills. 

o ask trainees to rate their beliefs or attitudes in specific areas that the 
training intends to change. 

o -observel trainees working with children to" see if'' they follow the 
practices and procedures the training emphasfees. 

o interview trainees , in depth to see^if they , really understand the 
essential concepts of the trainihg. * 

o give trainees realistic problems to solved to determine if they know how 
,to go about.it. • 

o rate the quality of written products oK other objects prpduced by 
.trainees that reflect their learning. * 

• . ' **, * 

• o ask trainees to rate their own learning or skills in specific areas, either 
in specific learning activities os^the program as a, whole. s 

. * o observe trainees in their own classrooms before and after training to 
determine if desirable changes in practice*occur v 

o analyze IEP's,. behavior charts, or dsfily logbooks to see if trainees are 
* ' implementing desirable practices. -# * i 

o. observe Qhil3ren # seryed/by trainees to see if their learnipg activity 
changes in desirable ways, such as increased time on task. 

. f 

o ask supervisors of trainees, parents, peers/ or children they serve to 
rate their ability and success. 

o look for changes in service delivery systems, resources^ or other 
contextual changes caused by trainees. 7 * ^ 

<• o identify highly successful, (or less successful) trainees and do case 

• j studies of what werft right (or wrong) ifi their training. „ ' 

jt o ask trainees l?ow useful specific aspects of the training were in their 
; jobs, or how well prepared they feel for their roles* , ' 

o folfowup trainees to find out if they $re employed in appropriate roles 
• and to determine how many children they serve. * 
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PLANNING YOUR_DEMONSTRATlON OF EFFECTS 



A demonstration of effects can be started at any point in the development .of a 
practice, though the best demonstrations are planned from the start of the 
developmental effort. ' To, get started, a simple plan can be developed in fhree 
steps: • 4 , • 

,1. Identify claims of effect iveness. A series *6f^ statements should be made 
about the effects of the practice — 1 effects, on trainees, effects on 
children, even effects on institutions. These claims should include 
statements about the direct, immediate effects of the practice $t\ those it * 
serves,- but may also include statements about indirect, long-range 
impacts. Potential claims can be derived from many\ sources — fronv 
proposals and other documents,^ from the values of. Staff, • from 
. participants, from funding agents, or from formal evaluations. 

2. Assemble existing evidence to support claims* Unless your practice is 
just beginning development, you will have some data collected for various 
purposes that may be used to demonstrate effects. Identify each kind of 
information that is already available anfl- review it carelully to see if.it 
can be used as. evidence to support any claims you wish to make. Also 
look at each claim, asking yourself if you 'have any evidence to support it. 

.3. Determine new evidence to be >% collected. Existing data collected for 
other purposes oftgn does not furnish sufficient evidence to support all of 
the claims you vtish to make. Review each claim in turn and ask yourself 
if the evidence you have convincingly supports it* If not, either restate 
the claim so that the evidence supports it,* or identify new evidence that 
can be feasible collected to support the claim as it is stated. Revise your 
evaluation plans and processes so that this evidence can be collected. * 

' As the developmental effort proceeds," this plan is periodically reviewed and 
refined to verify that ttfe final demonstration will be convincing. Prior to this review, 
a brief summary of claims and supporting evidence should be drafted, this interim 
demonstration may suggest revision of claims or future plans for the collection of 
evidence, and can also be disseminated to decision makers and otfjer audiences. Several 
sample effectiveness summaries are included in this package.. 

We have used this format for developing a plan for demonstrating effects: 
CLAIMS " % EVIDENCE WE HAVE ' EVIDENCE WE NEED 



3UDGING THE EVIDENCE - SHOULD -YOU BE CONVINCED? 



A demonstration of effects presents evidence designed xo convince some audience 
that a practice has achieved certain resuLts. To be convincing the demonstration must 
resolve three general questions, adapted from criteria, which appear in the Joint 
Dissemination Review P,anel Ideabook (1977)." 

Are there effects? - ■ ' ^ 

The evidence supporting each claim must be convincing. Two points are essential. 
First, the evidence. presented must be directly related to the claim to be made. For 
.example, if a program claims changes in classroom behavior after an Jnsem^g j?rpgr&mV 
then the evidence shpuld show that classroom behavior is different after me inservice, 
than it was before^ Evidence of changes in trainee beliefs, or even changes in child 
achievement do not support this claim. Second, the evidence tnust be, collected in 
reliable and, valid ways. Published instruments should match the claimed effect; 
instruments developed by the program should be tested before usej, multiply* data 
sources should be used wherever possible. Finally, appropriate statistical techniques 
should be used to demonstrate that the observed effects did not appear by chance. 

Is the effect due to* the practice? 

Effects can. be very real, but not necessarily caused by the' practice that is 
claimed to produce mem. A good demonstration rules out other plausible explanations 
for the observed effects or presents arguments showing how the effects must have 
resulted from the practice. For claims about '•child gains, a demonstration must show 
that normal maturation or the rest of the educational ^program 'did -not lead to the 
observed effects; appropriate use of coaj/ol groups may often be required for a 
convincing demonstration. For inservice education programs, a plausible clemonstrgtion 
should bhow 'that participants did not already have the skills; this often requires 
assessment prior to traioing. One' convincing technique is to show that the, program 
* feads to the claimed effects through a* series of direct, logical^ steps. For example, 
show that inservice training leads to specific changes in teacher classroom behavior 
which in turn, lead to specific changes in child learning behavior; then show that these 
.changes in learning behavior lead to improved achievement. 

Is the effect educationally significant? 

To be educationally significant, training effects should be related in some 'way to 
the .needs of trainees or children. In particular, personnel preparation practices aj*e 
educationally significant if they produce teacher abilities thart are related to teacher 
effectiveness. The effects demonstrated should also be important enough or large 
enough -to convince. -others that the results of the practice justify its cost. For 
example, a preservice program for teachers of the severely handicapped demosntrated 
that their graduates could apply behavioral management techniques. By citing research 
showing that such methods are essential to success with this population, this program 
also showed the educational significance of this skill. Demonstrating that a practice is 
*as good as, or even better than, the competition also helps to show educational 
significance. 
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A MODEL FOR DEMONSTRATING TRAINING EFFECTS 



Teacher educators often examine the effects .of *their training in a number of 
w,ays, such as knowledge tests, observation pf trainees by faculty, and ratings of 
graduates by supervisors. It is less common to see coherent arguments that trace 
training effects to their eventual impact on the learning of children, even though that 
is clearly the ultimate purpose of most training. The purpose of this model is to briefly 
sketch one way in which sUch an argument can be made; the content is adaptated from 
material originally presented hr Thompson (1981). 

i 

A brief examination of research on the relationships among the training of 
teachers, their classroom behavior, and the learning of children suggests a general 
framework for studying both the intermediate and the ultimate effects of teacher 
training. The idea behind this model is that such effects are more plausibly 
demonstrated if the argument is broken down into several intermediate steps between^ 
training activity and child growth. Each step is claimed to affect the next, and the' 
demonstration addresses the trut;h of each of these claims** 

THE MODEL * " 

cfuality 

teacher trainee 

training competence:— * effective 

program knowledge, classroom - ' ^improved 

skills & beliefs practices learning ^impact on , 

by trainees activity of child learning: 

children knowledge, 

skills & beliefs 



# , The provider of' teacher education directly controls only the activity that goes 
pn within the training program itself. The provider determines the content, the mode 
of its delivery, the types of feedback given to trainees, the physical setting, and other 
aspepts of the experience. Lectures, practica, classroom observations, seminars, 
workshops, peer work sessions, and individual assignments are some examples s of 
activities that are commonly used. 

It is claimed that participants in these activities gain certain knowledge, certain 
skills^ or certain beliefs by the time the {raining is completed/ These competencies are 
applied in the classrooms of trainees through the use of effective classroom practices. 
Such practices directly affect the mahy factors that impinge directly on' the child, such 
as curriculum materials, the physical setting in the, classroom, teacher instructional 
behaviors, or parent activities. 

Effective classroom practices lead, it is claimed, to changes in learning activity 
of the student. Such changes might include more books read, more time spent on 
learning tasks, completion of assignments, or x changes in study habits. Appropriate 
changes in the learning activity of children are claimed to improve learnings such as 
academic skills, social skills, motor skills, vocational skills, and the many other 
characteristics that educators seek to change. 
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As an example, consider a program which trains resource teachers. In 
demonstrating effects using this model, the jjsogram made the following claims: 

That training activities were appropriate to program' goals and were carried out as 
intended . The training program itself included a number of activities to develop skills 
in observation and charting, including both course content and supervised practice in 
field placements. Student Evaluation Sfale ratings of these experiences indicate that, 
in the judgment of participants, activities were carried out as planned. A review by an 
Advisory Committee included in Us findings a statement that program activities were 
appropriate to the goal of preparing competent resource teachers. 

That the competence of trainees improved in the desired direction as a result of 
participation in training^activities . Tests in several courses, a Teacher Practices 
£ttitude Survey, and the practicum evaluation process indicated that the the great 
majority of students understood observation methods, were skilled in techniques of 
observation and charting, and believed in the value of doing so. A review of entry 
assessment information indicated that few trainees possessed these skills on entry into 
the program. 

That trainees implement effective practices in their classrooms which are consistent 
wi th the training content . Surveys of graduates and of their supervisors indicate that 
most of those employed as resource teachers observed and charted some behaviors for 
each child in v their class. Structured * observation of the classes of a sample of 
graduates confirm these results. 

That such classroom practices affect the learning activity of children in positive ways . 
Previous studies reviewed by program facqlty indicated that the systematic observation 
and charting of child behavior leads to improved attention ,and increased time-on-task. 
This research finding: is confirmed by several studies within the program of trainee 
impact in practicum settings. 

That such changes in child learning activity lead to demonstrable positive impact on 
child learning .. The studies reviewed indicate that improvements in attention and 
time-on-task are directly related to high rates of change ia targeted academic and 
social behaviors. This relationship is also confirmed by studies within practicum 
settings served by trainees. 

In this brief ^xample, evidence from previous research is combined with 
information collected by the program itself to argue that the training of resource 
teachers in observation and charting is related to child learning. It should be clear that 
this argument does not conclusively prove that the training is the sole cause of the 
observed child learning and that it is seldom practical to do so. Training does not go 
on in isolation, but operates in a complex and value-charged context; many 
uncontrollable factors from this context affect each step in this model. These 
dimensions can intervene to break the chain of influence from teacher training to 
student at any point. The role of a demonstration of effect is not to provide definitive 
proof, but to provide plausible evidence of links at each step. 

/ 
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MEASUREMENT ISSUES IN DEMONSTRATING EFFECTS 



A great variety of claims can be made' about the effects of personnel 
preparation practices, _and an even* greater variety of evidence can be cited to support, 
such claims. Some of the more common general claims are listed below, and alternative 
ways to evidence them suggested. 



Claim: Trainees gain knowledge or cognitive skiils from the training. 

GOOD — Claims about knowledge and cognitive skill are often evidenced by 
various forms of testing. For, example, mo^t preservice courses include final 
examinations, and many inservice courses include tests as well. In addition to the usual 
formats for testing recognition and recall, problem solving and other higher cognitive 
skills can be assessed by a variety of methods which depend upon the specific skill. For 
example, skill in developing individual educational plans can be assessed in a 
paper-and-pencil simulation by presenting certain information and asking the trainee to 
set goals, write objectives, and identify appropriate instructional activities. Ratings of 
knowledge may also be collected from the trainee or from others in a position to make 
such judgments. 

BETTER — Evidence supporting claims about knowledge and other cognitive 
skflls is more convincing where the validity and accuracy of the measurement processes 
have been established. For example, items from knowledge tests can be pooled and 
their performance*studied under repeated use. The domains of knowledge to be assessed 
can be explicitly described, and items derived systematically from such a specification. 
Information from past uses of an instrument or from practicing professionals can be 
used to set appropriate criteria on which scores can be judged. Situations within which 
cognitive skills are assessed can be made more realistic, or actual work samples can be 
reviewed for evidence of such skills. Evidence? can not only be gathered from trainees, 
but can also be collected from graduates who are actually serving children. 



GOOD — Performance skills are often judged by program faculty or by other 
professionals such as supervising teachers. These judgments are often collected as 
ratings of skill on a modest number of broad areas, such as "able to manage behavior 
effectively in the classroom." Self-ratings of performance skills are also used, but may 
* be more indicative of self-concept ' than of actual level of skill. 



BETTER — Ratings of performance can be improved in a number of ways, 
including clearer specification of the skills to be judged, improved training of judges, 
and judging by several observers for .reliability. In some circumstances, the level of 
inference required by the observer can be reduced by a more detailed (or even 
behavioral) specification of the skill to be assessed. In other cases, existing observation 



developed instruments, providing a broader framework for interpretation of the results. 
Performance assessment can often be improved by expanding the time or the frequency 
of observation, or by observing in settings closer to the actual classroom. Perhaps the 
strongest evidence of performance skill is that collected from graduates on-the job, 
rather than from trainees in process. Information showing learning gains by children are 
convincing, if indirect, evidence of teacher skills. 



Claim: Trainees Win performance skills from the training. 



instruments that have been 




locally 
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Claim ; Trainees change beliefs or attitudes as a result of training. 



GOOD — Beliefs and attitudes are most often measured using self-reports of 
various kinds; trainees are asked to indicate their agree/nent with certain written 
statements, or are interviewed in a structured way. The general appropriateness of 
trainee attitudes may also be rated by others, such as a supervisor in a practicum. 

BETTER — Such measures of attitude and belief can often be improved by the 
usual psychometric procedures used to validate affective measures. The reliability of 
attitude survey instruments can be studied, and they can be given to practicing 
professionals to provide an additional reference point for interpreting results. 
Interviews can be structured and irfterviewers carefblly trained, to improve the quality 
of information they provide. ^ 

i ' 

Claim: Trainees are employed in positions appropriate 'to their training, providing 
appropriate services to handicapped children or making other worthwhile 
contributions. 

GOOD — Many preservice programs assist graduates in locating their first 
position, and if results are tracked, the program can summarize the positions first 
taken by graduates. Surveys of graduates often ask for position titles as well. 

BETTER — Such evidence can be improved through richer descriptions which go 
beyond position titles to some indication of the types of children served, the services 
provided, or the skills deployed. Followup .studies can be repeated over time to 
estimate attrition from teaching positions. Results of extensive mail-out surveys can be 
validated by telephone interviews or visits to samples of graduates. Programs need to 
make an active effort " to maintain contact with graduates in order to obtain such 
information at high rates of return, and such contact must also be reinforcing to 
graduates. This may require activities beyond data collection, such as maintaining an 
employment service, publishing a newsletter, or providing other information which will 
be of interest to the graduate. 

i 

Claim: Graduates implement effective teaching practices they were taught in their 
programs. 

GOOD — Surveys of graduates or of *their supervisors can ask specific questions 
about what goes on in the classroom. Such surveys have included written ratings, 
telephone interviews, or face-to-face conversations. : 

BETTER — General surveys can be improved by identifying and defining 
classroom practices that were emphasized in the training and focusing the survey on 
them. More convincing evidence of the use of effective classroom practices can be 
obtained through visits to graduates which include observations of classroom practice 
and in-depth .interviews on training and successful teaching. Such visits can be 
conducted with a sample or graduates and used to validate the results of self-report or 
supervisor surveys. 4 * 
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SOURCES AND RESOURCES on DEMONSTRATING EFFECTS 



Thompson, Elaine K. (ed). Using Student Change Data to Evaluate Inservice Education. 
Bloomington, Indiana: National Inservice Network, School of Education, Indiana 
University, 1981. 

" This report presents a general framework for looking at the effects of 
inservice education for teachers on the children these teachers serve. 
Pertinent evaluation questions about inservice education are mapped out, 
and the naturalistic and rationalistic paradigms for evaluation are 
compared. Seven distinct approaches are presented in detail. Summaries of 
effectiveness data for a few Inservice programs are included, as well as 
an annotated resource list. 7 

Duncan, J. R., OLsen, R^ M. & Schofer, R. C. Comprehensive System of Personnel 
Development: Evaluation Considerations. Columbia, Missouri: Department of Special" 
Education, University of Missouri, 1982. 

This report includes specific information on evaluation questions and 
information sources for each aspect of CSPD. It also includes general 
information of evaluation methods in .this context. 

# 

NOTE: ALSO MIGHT ADD TO THIS LIST - 

The JDRP Ideabook r 

The ETC Instrument catalog 

USOE material on validating achievement gains 

A brief note on the National Diffusion Network 

Dissemin/Action as a resource 

State Dissemination Systems (the IVD process) 
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